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SUMMARY 


The  Advanced  On-the-job  Training  System  (AOTS)  was  an  Air  Staff  directed,  Air  Force 
Human  Resources  Laboratory  (AFHRL)  developed,  prototype  system  which  designed,  developed 
and  tested  a  proof-of-concept  prototype  AOTS  within  the  operational  environment  of  selected 
work  centers  at  Bergstrom  AFB,  Texas,  and  Ellington  ANQB,  Texas,  from  August  1985  through 
31  July  1989.  The  System  Level  Testing  and  Evaluation-Lessons  Learned  paper  discusses  the 
need  to  closely  examine  the  constructs  used  to  measure  the  Testing  and  Evaluation  (T&E) 
objectives,  use  a  flexible  and  comprehensive  Master  Test  Plan  (MTP)  and  collect  background 
data  to  assess  its  influence  on  the  deployment  of  the  system.  Standardized  and  practical 
training  of  the  deployed  technology  will  facilitate  acceptance  and  use  of  the  system.  It  was 
also  found  that  the  optimal  compatibility  with  the  AOTS  was  with  participants  who  were  computer 
literate,  used  the  system  on  a  regular  basis,  and  were  involved  in  On-the-Job  Training  (OJT). 
An  additional  implication  of  the  T&E  is  to  keep  the  evaluation  and  analysis  practical  and  simple. 
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PREFACE 


The  purpose  of  this  paper  is  to  share  the  conceptual  insight  gained  from  the  formulations 
and  revisions  of  the  Master  Test  Plan  (MTP)  during  the  year-long  Testing  and  Evaluation  (T&E) 
of  the  Advanced  On-the-job  Training  System  (AOTS).  The  lessons  to  be  learned  from  this 
field-based  research  effort  will  center  on  the  major  measurement  objectives  -  the  Critical 
Issues  (i.e.,  Compliance,  Acceptance,  Suitability,  and  Performance).  Assumptions  underlying 
the  Critical  Issues  are  discussed,  along  with  their  usefulness  during  the  T&E,  and  Impact  on 
the  MTP.  In  addition,  this  paper  will  address  human  factors  which  proved  to  be  compatible 
with  the  acceptance  and  use  of  the  system.  Finaily,  this  paper  will  conclude  with  implications 
for  future  field-based  research  on  technology  deployed  in  the  operational  setting.  The  Air 
Force  Human  Resources  Laboratory  (AFHRL)  work  unit  number  for  the  project  is  2557-00-02. 
The  primary  office  of  responsibility  for  management  of  the  work  unit  is  the  Training  Systems 
Division  of  AFHRL.  The  Air  Force  AOTS  manager  is  Major  Jack  L.  Blackhurst. 
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ADVANCED  ON-THE-JOB  TRAINING  SYSTEM:  SYSTEM  LEVEL  TESTING 
AND  EVALUATION  -  LESSONS  LEARNED 


I.  INTRODUCTION 

For  over  10  years  the  Air  Force  has  recognized  deficiencies  in  the  existing  enlisted  On-the-Job 
Training  (OJT)  system.  The  Air  Force  Human  Resources  Laboratory  (AFHRL)  conducted  a  study 
of  the  OJT  system  in  1975,  and  the  Air  Force  Inspector  General  (IQ)  performed  a  functional 
management  inspection  (FMI)  of  OJT  in  1977.  Both  efforts  produced  similar  results:  the  OJT 
system  was  seriously  deficient  in  training  definition,  delivery,  evaluation,  administration  and 
personnel  utilization.  As  part  of  a  series  of  Air  Staff-directed  initiatives  designed  to  correct  these 
deficiencies,  a  follow-on  study  outlined  the  functional  and  automation  requirements  for  Air  Force 
OJT,  developed  a  system  specification,  and  picked  a  site  for  the  design,  development,  test,  and 
evaluation  of  an  Advanced  On-the-job  Training  System  (AOTS). 

Beginning  in  1985,  the  team  of  AFHRL,  Douglas  Aircraft  Company  (DAC),  and  Ball  Systems 
Engineering  Division  (BSED)  began  designing  the  prototype  AOTS  at  Bergstrom  AFB,  Texas.  A 
System  Level  Test  and  Evaluation  (SLT&E)  followed  the  successful  completion  of  the  development 
work  in  August  1988.  Over  600  Active,  Reserve,  and  Air  National  Guard  personnel  in  five 
different  specialties  (452X0,  454X0,  732X0,  751X1,  and  811 XX)  participated  in  the  12-month  test; 
thirty-four  work  centers  on  Bergstrom  AFB,  Texas,  and  Ellington  ANGB,  Texas,  were  equipped 
with  AOTS  computers. 

A  field-based  program  evaluation  approach  was  used  to  determine  if  AOTS  satisfied  four 
Critical  Issues  of  the  Master  Test  Plan  (MTP)  Compliance,  Acceptance,  Suitability,  and  Performance. 
While  the  first  issue  (Compliance)  addressed  the  design  and  specification  requirements,  the 
remaining  issues  focused  on  the  participants'  use  of  the  system.  More  specifically,  the  participants 
determined  whether  they  liked  AOTS  and  thought  it  was  easy  to  use  (Acceptance);  the  system 
solved  the  identified  deficiency  needs  of  the  existing  OJT  (Suitability);  and,  whether  AOTS 
improved  the  performance  of  individual  trainees  and  exhibited  the  potential  to  improve  the  whole 
OJT  system  over  time  (Performance). 

Overall,  the  results  of  the  Critical  Issues  were  favorable.  All  of  the  system  specification 
requirements  were  met  and  users  responded  favorably  toward  AOTS  and  felt  the  automated 
features  improved  the  efficiency  of  managing  and  conducting  training.  In  addition,  the  system 
demonstrated  the  potential  to  improve  task  knowledge  and  task  performance.  Additional  Testing 
and  Evaluation  (T&E)  results  are  beyond  the  scope  of  this  paper;  more  specific  information  can 
be  found  in  the  Final  System  Level  Testing  and  Evaluation  Results  Report  (Alba,  Gosc,  Blackhurst, 
Hays,  &  Marrero,  1990). 

This  paper  will  focus  on  the  conceptual  insight  gained  from  the  formulations  and  revisions 
of  the  MTP  during  the  year-long  T&E  effort.  Although  there  is  a  wealth  of  Information  to  be 
shared,  this  paper  will  limit  its  focus  on  the  basic  tenants  of  the  T&E-the  Critical  Issues. 
Assumptions  underlying  the  Critical  Issues  and  usefulness  of  these  issues  during  the  T&E  will 
be  discussed,  followed  by  their  affect  on  the  overall  MTP.  In  addition,  specific  information  on 
human  factor  variables  that  were  identified  during  the  deployment  of  the  AOTS  will  also  be 
addressed.  Finally,  a  list  of  implications  for  further  field-based  research  will  evolve  from  the 
concepts  reviewed. 

Measurement  Assumptions  of  the  Critical  Issuee 


Measurement  of  the  Critical  Issues  in  the  operational  setting  proved  to  be  a  major  challenge. 
For  the  most  part,  these  issues  were  measured  by  obtaining  the  participants'  responses  on  an 
opinion  survey.  Respondents  compared  AOTS  with  conventional  OJT  and  evaluated  the  impact 
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of  the  system  on  their  OJT  responsibilities.  Comparative  judgments  were  made  on  the  utility, 
efficacy,  efficiency,  and  relative  merit  of  the  ACTS.  To  make  these  judgments,  the  users  needed 
to  satisfy  one  of  two  basic  assumptions:  familiarity  with  either  the  AOTS  or  conventional  OJT 

This  assumption  is  based  on  the  understanding  that  to  judge  the  merit  or  relative  worth  of 
the  prototype  AOTS,  one  of  two  conditions  must  be  satisfied.  It  was  assumed  that  users  would 
have  a  basic  understanding  of  Air  Force  Regulation  (AFR)  50-23  and,  once  AOTS  was  deployed, 
it  would  be  used  on  a  regular  basis  to  conduct  OJT.  Neither  of  these  two  assumptions  were 
realized  to  the  level  expected.  Users  conducted  OJT  in  an  unstructured  manner  with  limited 
awareness  of  their  OJT  responsibilities,  and  the  system  was  underutilized. 

AOTS  was  designed  to  be  in  strict  compliance  with  the  Air  Force  OJT  standards  and  AFR 
50-23.  When  users  were  asked  to  judge  AOTS,  part  of  their  judgments  should  have  been  based 
on  how  well  AOTS  met  AFR  50-23.  Because  the  average  user  of  the  system  was  minimally 
aware  of  this  regulation  (beyond  completing  the  training  records--  623  forms),  there  was  an 
incongruity  in  judging  the  system  with  this  absolute  standard.  As  you  will  note  in  the  subsequent 
sections,  this  incongruity  affected  the  judgment  of  some  critical  issues  more  than  others. 

Given  the  limitation  of  this  standard  to  evaluate  the  AOTS,  there  are  several  questions  that 
will  be  addressed.  Namely,  How  useful  were  the  Critical  Issues  (Compliance,  Acceptance, 
Suitability,  Performance)  in  evaluating  AOTS?  What  revisions  were  made  in  the  MTP  to  better 
evaluate  AOTS  in  the  operational  setting?  What  human  factors  influenced  the  users’  judgment 
of  the  efficacy,  utility,  efficiency,  and  relative  value  of  the  AOTS. 

How  Useful  Were  the  Critical  Issues  In  Evaluating  the  AOTS? 

As  the  core  structure  of  the  SLT&E,  the  Critical  Issues  were  essential.  These  issues  served 
as  constructs-concepts  formulated  for  the  specific  purpose  of  observation  and  measurement 
(Kerlinger,  1973).  These  constructs  are  operationally  defined  to  assign  meaning  and  describe 
how  a  variable  (eg..  Critical  Issue)  will  be  measured.  Although  operational  definitions  are 
indispensable,  they  are  limited;  they  cannot  express  all  of  the  meanings  related  to  the  variable 
or  Critical  Issue.  Some  of  the  Critical  Issues  were  more  subjective  and  difficult  to  operationally 
define  than  others. 

The  Critical  Issues  could  be  viewed  as  a  hierarchy  of  least  to  most  difficult  to  measure, 
beginning  with  Compliance  and  followed  by  Acceptance,  Suitability,  and  Performance.  Compliance 
is  relatively  easy  to  measure  because  it  involves  the  objective  measurement  of  the  reliability 
and  stability  of  the  software.  Software  error  rate  and  response  time,  for  example,  were  objectively 
measured  and  rated.  However,  the  remaining  Critical  Issues  are  more  subjective  and,  consequently, 
more  difficult  to  operationally  define. 

The  Acceptance  Issue  Is  next  on  the  hierarchy  because  it  involves  the  basic  judgment  of 
whether  the  users  liked  the  system  and  felt  it  was  easy  to  use.  According  to  our  SLT&E  results, 
participants  who  used  the  system  on  a  regular  basis  liked  it.  Their  acceptance  was  directly 
related  to  the  regular  use  of  the  system.  The  other  remaining  issues,  however,  require  a  greater 
amount  of  differentiation. 

A  more  complex  issue  is  Suitability;  it  requires  two  levels  of  understanding.  Participants 
needed  to  know  Air  Force  OJT  and  learn  the  AOTS  to  judge  whether  the  system  meets  identified 
OJT  deficiencies.  In  some  cases  there  were  no  OJT  responsibilities  that  could  be  directly 
compared  to  an  AOTS  feature.  For  example,  due  to  time  constraints,  performance  evaluations 
are  typically  not  administered,  yet,  participants  were  asked  whether  these  evaluations  were  better 
for  assessing  task  proficiency.  As  previously  mentioned,  these  standards  of  judgment  were  not 
always  comparable  or  at  the  level  expected. 
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The  final  issue,  Performance,  ranks  as  the  most  difficult  concept  to  measure.  This  difficulty 
is  due  to  several  factors:  limited  amount  of  time  to  determine  performance  differences,  level  of 
expertise  required  on  the  system,  and  involvement  with  OJT.  Essentially,  opinion  survey  responses 
related  to  this  issue  involved  judgments  on  whether  AOTS  improved  the  quality  and  efficiency 
of  OJT.  In  order  to  respond  to  these  questions,  the  participant  had  to  compare  conventional 
OJT  methods  with  the  automated  features  of  AOTS  and  infer  the  differential  impact  on  training. 
Given  the  above  mentioned  constraints,  such  questions  as  determining  whether  AOTS  provided 
trainees  with  the  ability  to  perform  tasks  better  than  conventional  OJT,  or  improved  overall 
mission  effectiveness  was,  in  hindsight,  an  overly  ambitious  effort. 

As  constructs,  the  Critical  Issues  served  their  purpose;  they  provided  the  SLT&E  with 
meaningful  and  measurable  objectives.  However,  a  closer  look  at  these  constructs  reveal 
inherent  difficulties  with  the  underlying  assumptions  of  some  of  the  issues.  Consequently,  some 
issues  were  more  difficult  to  measure  than  others.  While  the  issues  of  Compliance  and  Acceptance 
were  easier  to  measure,  Suitability  and  Performance  (in  that  order)  presented  more  of  a  problem 
to  evaluate. 

What  Revisions  Were  Made  In  the  MTP  to  Better  Evaluate  AOTS  In  the  Operational  Setting? 

The  Master  Test  Plan  (MTP)  evolved  to  respond  to  the  limitations  of  the  last  three  Critical 
Issues-Acceptance,  Suitability,  Performance.  Borich  (1981)  makes  two  useful  distinctions  in 
understanding  the  evolvement  of  the  MTP.  Initially,  the  MTP  had  a  "value  oriented  emphasis." 
This  emphasis  involved  collecting  and  evaluating  data  on  the  basis  of  previously  established 
criteria  for  the  program  (e.g.,  Critical  Issues).  As  the  MTP  evolved,  however,  it  became  more 
flexible  and  comprehensive.  Multifaceted  data  sources  and  supplementary  evaluation  efforts  (e.g., 
Performance  Study,  individual  interviews)  resulted  in  a  more  global  assessment  of  the  OJT 
setting  of  which  AOTS  was  a  part.  It  evolved  into  a  "systems  oriented"  approach,  evaluating 
AOTS  as  a  part  of  the  ecological  constraints  and  expectations  that  influence  the  OJT  environment. 
This  approach  evaluated  the  extent  to  which  field  activities  and  organizational  factors  affected 
the  outcome  of  the  prototype  and  vice  versa. 

The  more  flexible  MTP  assessment  approach  included  the  collection  of  process  and  background 
data  (see  MTP  for  more  specific  descriptions).  This  flexibility  resulted  in  the  use  of  subsamples 
representing  varying  levels  of  usage  of  the  AOTS.  For  example,  a  subsample  of  higher  frequency 
users  was  identified  and  interviewed  to  obtain  more  valid  feedback  on  the  prototype.  Process 
data  consisted  of  standardized  observations  conducted  on  a  regular  basis  at  the  work  center 
level,  documentation  of  information/assistance  calls  ("help  hotline"),  and  computer  records  of 
system  use  by  individual  user  Identification  number.  In  addition,  a  log  of  various  activities 
during  the  deployment  period  was  maintained.  These  supplemental  measures  expanded  the  MTP 
to  include  information  on  the  reciprocal  influences  of  the  prototype-user-environment.  The  final 
MTP  provided  a  more  comprehensive,  flexible  and  multifaceted  assessment  of  automated 
technology  in  the  Air  Force  OJT  operational  setting. 

Human  Factors  Related  to  Outcome  of  SLT&E 


An  overall  pattern  of  the  SLT&E  results  revealed  that  the  motivational  aspects  of  learning 
and  using  the  system  can  be  described  by  the  "Goodness  of  Fit"  model.  Although  this  model 
does  not  encompass  all  the  possible  factors  that  Influenced  the  use  of  the  system,  it  provides 
a  heuristic  model  for  understanding  the  SLT&E  results.  According  to  this  model,  the  acceptance, 
utilization,  and  overall  compatibility  with  the  system  was  contingent  on  several  user  characteristics. 
These  characterlstics-computer  literacy,  regular  use  of  the  system,  and  involvement  in 
training-were  identified  during  the  T&E  and  wilt  be  addressed  individually. 

The  participants'  familiarity  with  computers  provided  a  greater  likelihood  of  system  acceptance 
and  use.  Based  on  work  center  observations,  it  was  noted  that  some  of  the  participants  had 
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limited  experience  with  computers  and,  consequently,  were  hesitant  to  use  the  system.  These 
individuals  were  described  as  "computer  phobic."  Despite  the  one-to-one  training  and  the 
user-friendliness  of  the  system,  some  of  the  participants  were  afraid  to  make  mistakes  for  fear 
of  damaging  the  training  records.  In  contrast,  those  individuals  who  were  computer  literate  or 
not  intimidated  by  computers  were  more  willing  to  learn  and  use  the  system. 

Once  an  individual  received  training  on  the  system,  it  was  important  to  maintain  a  level  of 
regular  use  of  the  system.  Regular  use  would  allow  a  greater  level  of  familiarity  and  ease  in 
applying  the  various  automated  functions  to  complete  OJT  responsibilities.  Participants  who 
used  AOTS  on  a  regular  basis  became  proficient  on  the  system.  Their  motivation  to  learn  the 
AOTS  was  linked  to  their  involvement  and  need  for  the  automated  functions. 

During  the  SLT&E,  background  training  characteristics  of  participants  who  responded  both 
favorably  or  unfavorably  to  the  AOTS  on  the  opinion  surveys  were  analyzed.  A  major  distinction 
among  those  who  responded  favorably  was  their  involvement  with  training.  They  had  an  average 
of  four  to  five  trainees  during  the  deployment  period  as  opposed  to  one  or  no  trainees  among 
the  less  favorable  respondents.  In  addition,  Air  Force  Reserves  supervisors,  responsible  for 
maintaining  weekend  personnel  training  records,  were  equally  positive  of  the  automated  AOTS 
functions.  Needless  to  say,  the  greater  the  need  the  higher  the  motivation  to  learn  the  system. 

All  of  these  characteristics  provided  a  good ness-of -fit  between  the  user  and  the  AOTS.  if 
the  participant  was  not  computer  phobic  or  had  some  computer  experience,  spent  time  learning 
the  system,  and  was  Involved  in  OJT,  he  or  she  was  likely  to  be  favorable  toward  AOTS. 
Although  these  characteristics  are  not  meant  to  be  all  inclusive  of  those  responding  favorably 
toward  AOTS,  the  optimal  user-system  compatibility  was  with  participants  that  exhibited  all  of 
these  characteristics. 

Implications  for  Future  Field-Based  Research 

The  SLT&E  of  the  AOTS  provided  an  opportunity  to  conduct  research  in  the  operational 
OJT  setting.  Given  the  minimal  research  on  Air  Force  OJT  this  field-based  research  effort 
provided  the  unique  opportunity  to  understand  Air  Force  OJT  and  the  impact  of  the  technology 
offered  by  the  AOTS.  Throughout  the  project,  the  major  emphasis  was  to  measure  the  impact 
of  this  system  without  being  too  intrusive  and  interfering  with  mission  priorities.  Overall,  the 
challenge  of  evaluating  this  technology  among  the  constraints  and  background  characteristics 
of  the  operational  setting  was  an  informative  and  educational  experience.  The  following  is  a  list 
of  implications  of  this  effort  that  may  be  useful  In  future  field-based  research. 

1.  The  development  of  the  MTP  should  be  preceded  by  a  comprehensive  evaluation  of  the 
operational  setting.  This  evaluation  should  not  be  limited  to  identifying  deficiencies  but  include 
the  assessment  of  various  factors  that  contribute  to  these  deficiencies.  Environmental  events 
and  constraints  that  influence  the  operational  setting  and  unique  training  differences  among  the 
various  specialties  should  be  evaluated.  This  information  would  contribute  to  planning  decisions 
in  developing  the  prototype  system.  It  would  also  provide  a  realistic  awareness  of  the  assumptions 
underlying  the  program  evaluation  objectives. 

2.  The  SLT&E  should  include  an  MTP  that  is  both  flexible  and  multifaceted.  Evaluation  efforts 
should  be  implemented  prior  to  and  during  the  deployment  of  the  technology.  This  approach 
would  allow  comparative  baseline  data  and  information  that  may  result  in  modifications  of  the 
T&E  efforts  as  needed.  In  addition,  information  collected  during  the  deployment  can  be  useful 
in  understanding  and  interpreting  the  outcome  data. 

3.  A  structured  training  program  Is  necessary  to  facilitate  the  learning  of  the  system. 
Standardization  of  the  training,  use  of  visual  aids,  individual  or  small  group  training  (no  more 
than  three  people),  and  follow-up  training  are  important.  Emphasizing  the  most  pertinent  functions 


specific  to  the  individuals'  OJT  responsibilities  would  provide  a  level  of  practicality  to  the  learning 
process.  In  addition,  certification  on  the  system  based  on  demonstrated  competency  should  be 
documented  as  part  of  the  individual's  training  record.  This  recognition  would  highlight  the 
training  and  enhance  the  motivation  to  learn  the  system.  Moreover,  it  would  help  identify 
individuals  who  have  learned  the  system  and  could  provide  valid  feedback  during  the  SLT&E. 

4.  Include  an  ecological  or  systems  approach  as  part  of  the  SLT&E.  This  approach  allows 
the  ongoing  collection  of  background  factors  and  events  that  have  an  impact  on  the  deployment 
of  a  system  and  vice  versa.  It  is  just  as  important  to  be  knowledgeable  of  how  the  ecological 
constraints  and  background  factors  affect  the  deployment  of  a  prototype  system  as  it  is  to 
know  the  impact  the  system  has  in  the  work  setting. 

5.  The  KISS  principle  ("keep  it  sufficiently  simple")  is  important.  Everything  from  the 
methodology,  statistical  analysis,  and  communication  of  results  should  be  screened  for  practicality, 
accuracy,  clarity,  and  brevity.  There  should  be  justification  for  all  the  data  collection  and  every 
effort  should  be  made  to  collect  data  in  an  unobtrusive  manner.  Given  the  experimental  constraints 
of  field-based  research,  the  use  of  inferential  statistics  should  be  second  to  descriptive  data. 
In  most  cases,  results  are  best  communicated  and  understood  using  basic  descriptive  statistics 
such  as  percentages,  averages,  and  range  of  scores.  The  use  of  graphs  to  depict  the  T&E 
results  is  an  art;  it  is  a  powerful  tool  in  conveying  results  in  a  clear  and  succinct  manner. 

In  summary,  it  could  be  argued  that  these  implications  are  applicable  to  assessing  the 
deployment  of  technology  in  other  field-based  research.  Programmatic  efforts  to  initiate  change 
in  the  operational  setting  should  be  sensitivity  to  the  influence  of  contextual  constraints  and 
background  factors.  A  systems  oriented  approach  emphasizes  the  need  to  measure  the  impact 
and  reciprocal  influences  of  these  environmental  factors  prior  to  and  during  the  deployment  of 
technology  in  the  operational  setting.  Standardized  training  is  recommended  to  assure  a  level 
of  competency  and  acceptance.  Given  the  rapid  growth  and  availability  of  technology,  it  is 
likely  that  there  will  be  demands  for  field-based  program  evaluation  efforts  similar  to  that  used 
with  the  AOTS. 
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